User interface system based on pointing device

ABSTRACT

The user interaction system comprises a portable pointing device ( 101 ) connected to a camera ( 102 ) and sending pictures to a digital signal processor ( 120 ), capable of recognizing an object ( 130 ) and a command given by the user ( 100 ) by moving the pointing device ( 101 ) in a specific way, and controlling an electrical apparatus ( 110 ) on the basis of this recognition. (Characteristic) pattern generation means ( 116 ) can be included in the system for facilitating object recognition.

The invention relates to a user interaction system, comprising:

an electrical apparatus;

a portable pointing device operable by a user for pointing to a regionin space;

a camera taking a picture; and

a digital signal processor, capable of receiving and processing thepicture, and capable of transmitting user interface information derivedfrom the picture to the electrical apparatus wherein the camera isconnected to the pointing device so that in operation it images theregion pointed to.

The invention also relates to a pointing device for use in a userinteraction system.

The invention also relates to an electrical apparatus for use in theabove mentioned user interaction system.

Such a system is known from D1 (DE10110979A1). D1 relates to anarrangement that has a camera that can recognize objects by taking asample picture of it and by comparing this picture with a reference.Once an object has been recognized the arrangement can send controlcodes (such as conventional Remote Control codes) to equipment,associated with the recognized object, to be controlled. A user can begiven feedback, by the arrangement, upon object recognition (light,audio, haptic). The inventor realized that it is a disadvantage of D1that its recognition of an object is solely dependent on the object asis found in real life. The capability of the system of D1 to recognizean object may therefore be poor.

Such a system is partly also known from US-A-2001/0030668. This documentdescribes a universal remote control system in which a user marks alocation on a display with a pointing device displaying a recognizablecharacteristic, e.g. a laser pointer emitting a red spot, and a cameratakes a picture of the display and calculates to which location the userhas pointed. An electrical apparatus, e.g. a computer or a television,then performs an action corresponding to the marking by the user. E.g.if an icon on the display is marked by flashing the laser pointer twice,a corresponding computer program is started.

It is a disadvantage of the known system that the system is tailored fora particular display. E.g. the display may be a projection from an LCDbased front projector on a screen, to which projector a laptop isconnected. The camera in the known system has such a position andviewing angle that it is capable of imaging the screen. Next, the DSP istrained to discriminate an area in pictures from the camera, whichcorresponds to the location of the screen. In the prior art this can bedone in a simple way. First a picture of the screen is captured bothwith and without illumination with a picture from the projector. Bycomparing the difference of those two pictures, the area of the displayin space as viewed by the camera is obtained. Second a perspectivetransformation is derived which determines the relation between alocation pointed to on the display and a location in the picture forprojection, in the frame buffer of the laptop. The known document alsodescribes pointing to a virtual display, e.g. a wall, by first manuallyselecting the corners of the virtual display. However the system isalways calibrated for one display and in case a user wants to interactwith another display, he has to set up the system again, or use a secondsystem.

It is a first object of the invention to provide a user interactionsystem of the kind described in the opening paragraph that improvesobject identification.

It is a second object of the invention to provide a user interactionsystem of the kind described in the opening paragraph in which it isrelatively easy to interact with a number of available electricalapparatuses, more particularly displays.

It is a third object of the invention to provide a pointing device ofthe kind described in the opening paragraph which makes interaction witha number of available electrical apparatuses relatively easy.

It is a fourth object of the invention to provide electrical apparatusfor use in the above mentioned user interaction system.

The first object is realized in that the system further comprises apattern creation means to exhibit a characteristic pattern that iscreated for identification purpose by the pointing device of the patterncreation means associated apparatus.

The second object is realized in that the camera is connected to thepointing device so that in operation it images the region pointed to. Afixed camera only images a certain region of space. It is possible thatin the known user interaction system, the user's pointing device, or theobject pointed to are not even in the field of view of the fixed camera.It is a major advantage of a pointing device with an attached camera,e.g. on the front end of the pointing device closest to the regionpointed to, that the camera is always imaging the region pointed tosubstantially optimally. The object pointed to is in general theelectrical apparatus to which the user interface information, e.g.apparatus control data, is sent, but can be any object present in theroom in which the user resides, as long as it can be recognized by theDSP. The apparatus to be controlled is then identified e.g. by pointingto it previously.

In an embodiment the user interface information comprises apparatuscontrol data for controlling operation of the electrical apparatus.There can be many apparatuses present in a living room which e.g.produce sound. Each of these devices has a control for increasing ordecreasing the sound volume. By pointing the camera of the pointingdevice towards a particular apparatus and performing a “volume up”command, e.g. by pressing a button on the pointing device or making aspecific movement with the pointing device, the sound volume of thedevice pointed to will increase. Using a known universal remote controlcan have as a disadvantage e.g. that an apparatus responds which was notintended by the user, because it uses the same remote control code.

It is advantageous if the digital signal processor comprises an objectcharacterizing means for characterizing an object or part of the objectpresent in the picture of the region imaged by the camera, by providingfirst object characterizing features to a comprised objectidentification means for identifying the object, and which objectidentification means is capable of outputting object identification datafrom which the user interface information is constructed.

In case different electrical apparatuses are to be controlled, they allhave to be recognized automatically. The DSP can be e.g. a genericprocessor running algorithms for characterizing the apparatus, or it canbe an ASIC. Typically the object characterizing means will performimage-processing algorithms on a picture of the apparatus. E.g. therelative dimensions of the apparatus can be calculated and the numberand positions of buttons on the apparatus can be analyzed. The objectidentification means associate with these resulting first objectcharacterizing features a specific apparatus, e.g. from information inan apparatus table. Other information about the apparatus, e.g. “firsttelevision produced by company X with serial number Y” may be present,like the apparatus control data that the apparatus understands.

It is also advantageous if the digital signal processor comprises:

motion trajectory estimation means for estimating a motion trajectory ofthe pointing device and outputting a first motion characterizingsignature, a signature being a mathematical abstraction of the motiontrajectory; and

signature identification means for identifying the first motioncharacterizing signature and outputting command identification data,which represents a user interaction command, corresponding with thefirst motion characterizing signature, from which command identificationdata the user interface information is constructed.

To avoid the need of a large number of buttons on the pointing devicefor all the different commands that can be sent to all the differentapparatuses, and to make the user interaction system more user friendly,it is advantageous if at least some of the apparatus control data isgenerated on the basis of movements of the pointing device by the user,which pointing device is typically in his hand. The signature of anupwards movement could mean “volume up”, but in contrast with a standardremote control, the amount of volume change can according to theinvention depend on the brusqueness of the motion. If the user moves hisarm up slowly, the volume should e.g. change only slightly and incontrast if he moves his arm up fast, the volume should go up a lot.Determination of the motion of the pointing device can be done on thebasis of motion sensing means, e.g. a mass on a deformation sensor, agyroscope, differential GPS, etc. However since a camera is alreadypresent, the motion can also be determined by imaging successivepictures and applying a motion estimation algorithm. Since global motionestimation is simpler than precise motion estimation of multipleobjects, the camera can be a simple CMOS based camera or even atwo-dimensional array of a small number of light sensitive elements.

It is also advantageous if the digital signal processor comprisesidentification improvement means, which are capable of further improvinga probability that the object represented as object identification data,and user interaction command represented as command identification data,are more reliably identified based on predetermined rules, yielding morereliable user interface information. Since identification of the objectand motion signature need not always be perfect, given the variabilityof conditions in a room, identification improvement means can beincorporated which e.g. apply an artificial intelligence set of rules onthe identified object identification data and command identificationdata, possibly also taking into account the characterizing featuresfirst object characterizing features and first motion characterizingsignature. E.g., a rule for determining the apparatus pointed to cantake into account the room in which the pointing device is present, orthe time of the day. E.g. the first thing what a user may typically doif he comes home and picks up the pointing device is to switch on thelights. His intention can be verified by the DSP on the basis of apicture of these lights.

It is further advantageous if the predetermined rules compriseprobabilistic calculation of the likelihood of an {object identificationdata, command identification data}—pair, taking into account at leastone of the following a priori known information units {room in which thepointing device resides, previous command issued by user, statisticalfrequency that a user issues a particular command and time of the day}.E.g. the likeliness of a command can be calculated based upon astatistical profile of which commands a user usually uses, and basedupon previously issued commands. Also, the likelihood of an apparatuscan be updated based on the room in which the pointing device ispresent. On Friday at eight o'clock the user might want to watch hisfavorite television program.

It is also advantageous if the digital signal processor comprises objectassociation means for providing to the object identification meansobject association data comprising at least one of the data entitiesbeing: associated object characterizing features and object relateddata—,

the object association data being derivable from object template data inobject memory originating from at least one of the methods:

the object template data is obtained from object training meansperforming a predetermined calculation on second object characterizingfeatures outputted by object characterizing means; and

the object template data is derived from inputted object data.

The user interaction system is much more versatile if, instead of beingpre-constructed by the manufacturer for controlling a fixed number ofapparatuses, the user can adapt it to his changing environment. E.g. ifhe buys a new apparatus, he would like this apparatus to be controllableby means of the same system. Also if he buys a calendar, hangs it on thewall, and points to it with his pointing device, he would like an actionto occur, e.g. starting up a calendar tool on his p.c. For this objectassociation means are required. The user images a new apparatus, e.g.under a few different angles. A number of image processing algorithmsrunning on the object characterizing means extract features for thisobject which are stored as an object template, possibly after firstcalculating invariant apparatus specific features. He then selects witha selection user interface which apparatus corresponds to this objecttemplate. This is a so-called training phase of the system. When laterthe system is used in a so called command phase, for sending apparatuscontrol data to an identified apparatus, the object identification bythe object identification means can be aided by means of the data storedin the object association means. E.g. on the basis of the objecttemplate data a number of associated object characterizing features,corresponding to what the object characterizing means output as firstobject characterizing features, can be sent to the object identificationmeans, which then performs a comparison, selecting the correctidentified apparatus. These features can also be preloaded in the objectidentification means. The object association means can also send objectrelated data to the object identification means, comprising at least anidentifier of the apparatus, and possibly further data such as supportedcommands. In an alternative realization, the object association meanscould also perform the object identification itself, in which case onlyobject related data is sent to the object identification means. To avoidthat the user has to input a lot of information through the selectionuser interface, object related data can also be inputted through anobject data input, e.g. directly from an apparatus to be identified andcontrolled. Via this input, characteristic object template data can alsobe inputted, e.g. the manufacturer of an apparatus might provide apicture taken under well-controlled imaging conditions.

It is also advantageous if the digital signal processor comprisessignature association means for providing to the signatureidentification means signature association data—comprising at least oneof the data entities being: associated signature features and commandrelated data—,

the signature association data being derivable from signature templatedata in signature memory originating from at least one of the methods:

the signature template data is obtained from signature training meansperforming a predetermined calculation on a second motion characterizingsignature outputted by the motion trajectory estimating means; and

the command template data is derived from inputted command data.

According to a similar rationale, signature association means can becomprised to enable the training of new motion trajectories applied tothe pointing device by the user and characterized as signatures. Inthese realizations command data is inputted e.g. by an apparatus whichprovides a menu of its supported commands.

It is also advantageous if the first motion characterizing signature isderived on the basis of successive pictures imaged by the camera atrespective instances of time. Since a camera is attached to the pointingdevice, imaging its every move, and global motion can be estimated by asimple motion estimation algorithm, successively imaged pictures can beused for deriving the intended user interface command.

The second object is realized in that the pointing device comprises acamera and is capable of sending a picture to a DSP. As alreadydescribed above the user interaction system becomes more versatile withsuch a pointing device.

An embodiment of the pointing device is characterized in that it iscapable of sending a picture to the digital signal processor, which iscapable of sending user interface information to an electrical apparatusbased on the picture

In an embodiment the DSP is comprised in the pointing device. Thepointing device can then also operate separate from the user interactionsystem. E.g. the pointing device can be taken to a supermarket, processimaged pictures and store the corresponding first object characterizingfeatures. Control actions by home apparatuses can then be performed at alater time.

It is advantageous if the pointing device comprises motion sensing meansfor sensing a motion trajectory of the pointing device. Irrespective ofwhether the device is used for recognizing objects, it can be used tosend apparatus control data corresponding to specific movements by theuser. The intended apparatus in such an application of the pointingdevice could e.g. be fixed or indicated with a button.

It is advantageous if the pointing device comprises a characteristicprojector for optically projecting a characteristic pattern towards aregion pointed to. It is advantageous if the user gets a feedback ofwhich apparatus or part of an apparatus he is pointing to. Otherwise, ifhe is not aiming correctly, he might e.g. virtually push the wrongbutton of the apparatus. One form of feedback could be a small pictureon a display of the pointing device which shows the center of the regionimaged by the camera. An apparatus or button on an apparatus in themiddle of the display is the one intended. Another form of feedbackcomprises projection of a characteristic pattern, e.g. a white cross, ora laser dot, which illuminates the intended part of an apparatus. Morecomplex patterns such as a two-dimensional grid of lines can aid the DSPin identifying the geometry of the imaged apparatus.

It is advantageous if the pointing device comprises a programmable userinterface code generator and a wireless transmitter for transmitting thecode to the electrical apparatus. Instead of the DSP sending apparatuscontrol data to the apparatus, e.g. by means of a wireless home networkbased on Bluetooth, the pointing device itself may directly send theappropriate remote control command, e.g. by means of infrared radiation.The advantage compared to a classical universal remote control is thatthe DSP automatically identifies the apparatus, from a camera picture,and generates the correct code.

It is advantageous if feedback means are present for feedback of userinterface information. E.g. a display is useful. Another feedback canprovide some kind of force feedback, e.g. by means of a mass which ismoved. This can e.g. simulate tipping against the screen of a TV.

The third object is realized in that interface means are comprised whichallow the apparatus to send information about supported commands to apointing device as claimed in claim 1, based on an “identify supportedcommands” call of the pointing device to the apparatus. The DSP or partof the functionality of the DSP might also be comprised in theapparatus. The apparatus might also transmit object characterizingfeatures and so on.

The invention will be apparent from and elucidated with reference to thefollowing description and the accompanying drawing, showingschematically examples of components of the user interaction system andpointing device according to the invention. In this drawing:

FIG. 1 schematically shows a room in which a user interaction systemaccording to the invention is present;

FIG. 2 schematically shows a digital signal processor according to theinvention;

FIG. 3 schematically shows a pointing device according to the invention;and

FIG. 4 shows two examples of a motion trajectory and correspondingsignature.

In FIG. 1 a room is shown containing various electrical apparatuseswhich can receive user interface information I, e.g. electricalapparatus 110 being a plasma display, but also a personal video recorderor a telephone or a washing machine can be present, with which a user100 can interact. The room also contains various objects, beingelectrical apparatuses or just any physical object like e.g. a window,or object 130 being an audio apparatus, vase 170 and electricalapparatus 110, that can be pointed to by means of a pointing device 101.

According to the invention, the pointing device 101 contains a camera102, and can send pictures of regions of a room or objects in thoseregions to a digital signal processor (DSP) 120, which can identify theregions or objects on the basis of one or more pictures imaged by thecamera 102. The camera is connected to the pointing device 101 in such away, that it images well the region pointed to. E.g. it can typicallyreside at the far end of the pointing device 101, but it could also bemounted on the side under an angle. The user 100 has the freedom topoint to whatever object he wants, and in such a way a veryuser-friendly and powerful user interaction system can be realized.

The DSP 120 can be e.g. a home computer controlling apparatuses in ahome network, or it can be incorporated in the pointing device 101. Thecamera 102 can be a CCD camera or CMOS camera. The connection betweenthe camera 102 and the DSP 120 can be wireless if the two are separate.The pointing device 101 can already perform part of the algorithms to beperformed by the DSP 120 for identification of an object, so that e.g.only a small set of features need to be sent to the external DSP 120 forfurther processing. In fact the functionally shown DSP 200 can bedistributed in a number of parts in the pointing device 101, room,different apparatuses and so on.

The DSP 120 is designed to send user interface information I, e.g.apparatus control data ac, to an identified apparatus. E.g. user 100 canpoint the pointing device 101 to light 160 and push an on-button on thepointing device 101, which results in the DSP 120 sending an on-commandto the identified light 160. The object identified needs not be theapparatus to be controlled itself. E.g. pointing at vase 170 may start aparticular program on a p.c., the output of which is shown on thedisplay apparatus 110. Parts of an object may be pointed to, e.g.pointing at a volume button 134 of object 130 increases or decreases thevolume of this audio apparatus, in dependence on an additional actionsuch as the pushing of a button or a movement of the pointing device101. Pointing to the channel selection display 132 and making a sidewardmovement may change the channel. The DSP 120 may also recognize thestatus of a part of an object, e.g. that cassette loader 136 is leftopen, and issue a “close” command. Other specific situations can also bedetected or monitored. User 100 may also point to a piece of furniture140 or a part 142 thereof. If the piece of furniture 140 is a cupboarddifferent actions can result from pointing to different drawers. If thepiece of furniture 140 is an intelligent refrigerator, pointing to itcan result in the user defined action of ordering beer over theInternet. Apart from apparatus control data ac, other user interfaceinformation I can be sent, e.g. picture data pd. E.g. first electronicpainting 150 and second electronic painting 152 display a variablepicture. They can be realized with electrophoretic electronic inkdisplays. User 100 can capture the picture displayed on the firstelectronic painting, or even a picture on a calendar hanging on thewall, and transfer it as picture data pd to the second electronicpainting 152, possibly after some image processing such as e.g.perspective correction.

FIG. 2 schematically shows functional components in a digital signalprocessor 200—an embodiment of 120 in FIG. 1. Not all componentsdescribed are necessarily present in a particular embodiment of the userinteraction system. The DSP 200 can be realized as a generic processor,an ASIC, or separate components on a circuit board. The DSP 200 mayreceive camera input c and sensor input s, e.g. from motion sensingmeans 304. The DSP 200 may comprise an object characterizing unit 202for characterizing a picture of an object 130 or part of the object 130as imaged by the camera 102. The object characterizing unit 202 iscapable of providing first object characterizing features f1 to anobject identification unit 204. The first object characterizing featuresf1 may be of different kinds depending on the image processingalgorithms embodied in the object identification unit 204. Typically theobject is first isolated from the remaining parts of the imaged picture.This can be done e.g. on the basis of edge and curve detection on thepicture. Another option is 3D analysis of the imaged region, e.g. with aprojected grid or a second camera. Part of an object may be detected onthe basis of correlation techniques, and then the rest of the object canthen be verified.

Second the first object characterizing features f1 are calculated. E.g.a contour of the object and its composing parts can be characterized bymeans of a curvature versus angle characteristic. Or the ratiowidth/height may be calculated. Also, the isolated object region itself,or a transformation of it may be used, typically after first performinga perspective correction.

The object identification unit 204 then identifies the object on thebasis of the first object characterizing features f1 received from theobject characterizing unit 202. E.g. any known classification techniquein multidimensional feature space might be used. The output of theobject identification unit 204 is object identification data oi, whichcan be a simple number or a data structure containing additionalinformation about the identified object. The characterizing features andfurther data of a number of apparatuses might be e.g. stored a priori inan object characteristics memory 230 by the manufacturer of the pointingdevice containing the DSP 200, or might be preloaded from internet ifthe DSP 200 is a computer. Instead of identifying an object, the DSP mayalso identify a region of the room. E.g. pointing the pointing device101 towards the ceiling may issue the command “lights on”, whereaspointing to the floor switches the lights off. Or pointing to one of thecomers may invoke an associated command. The corners may be identifiedon the basis of objects present, e.g. on the left side a plant and onthe right side a cupboard.

The user interaction system is much more useful however if a user cantrain new objects, such as vase 170, himself. Therefore an objectassociation unit 212 might be comprised, which provides the objectidentification unit 204 with object association data oad. The objectassociation data oad might comprise characterizing features fa of anobject, similar to the first object characterizing features f1, based onan object template. The object template can be obtained by applying atraining phase. A new object is imaged by the camera 102 under a numberof conditions, e.g. different angles, lighting etc. The objectcharacterizing unit 202 generates second object characterizing featuresf2 for all the pictures. An object training unit 221 in the objectassociation unit 212 generates an object template corresponding to theobject which is stored in object memory 220. The template can e.g. bethe cluster of second object characterizing features f2 of all thepictures, or average features, or some invariant feature calculated onthe basis of the variable second object characterizing features f2. Theobject template might also be derived by the object training unit 221 onthe basis of characterizing features coming in from the outside asobject data od. This object data might originate from an apparatus whichinputs e.g. pictures of it taken from different angles by amanufacturer. The object data od and object association data oad mightalso comprise object related data id, i.e. all kinds of informationconcerning an apparatus, like e.g. supported commands.

To facilitate object recognition, certain objects like e.g. apparatus110 may display a characteristic pattern 116, e.g. they may flash a redcross a predetermined number of times with different on/off intervals,in response to an “object identify” call of the pointing device. Theapparatus 110 might also comprise an interface unit 190, which makes itmore useful in a user interaction system according to the invention.E.g. an apparatus like an internet radio can have a “service discovery”function which communicates to the pointing device 101 which commands itsupports. It might even transmit infrared remote control sequencescorresponding with these commands, and so on. This facilitates thetraining of the DSP 120, 200 by the user 100. E.g. in response to thecommunication, the commands can appear as a menu of predefined icons ona display 316 of the pointing device 101. The interface unit 190 canalso be realized separately. E.g. a washing machine might not have anintricate remote control facility. A plug-on box can be provided whichreceives standard commands from the DSP 200 and interfaces in a simpleway with the apparatus, e.g. starting it.

The DSP 200 might also comprise components for analyzing gestures madewith the pointing device 101 by the user 100. For this, the motiontrajectory 400 in FIG. 4 of the pointing device 101 is used, which iscalculated e.g. on the basis of pictures from the camera 102. FIG. 4 ashows an upward motion trajectory 400 in a three-dimensional coordinatesystem 404. It is summarized by signature 402 being a mathematicalrepresentation of that upward motion trajectory. The signature 402 canbe linked, e.g. by means of a command table, with command identificationdata ci, e.g. in this case the user 100 might reserve this motion for a“volume up” command. A circular motion trajectory 410 and circularsignature 412 might mean to one user “rewind”, whereas another user hastrained the DSP 200 to relate this circular signature 412 with a “deviceon” command. The user can e.g. roughly orient his pointing device 101towards light 160 and make the “device on” motion, on the basis of whichthe DSP 200 switches the light 160 on. Alternatively the user 100 canalso make the “device on” motion somewhere in the air and then point tolight 160, or a television or whatever object he wants to be switchedon. The signatures 402, 412 can be parameterized in any way and matchedby any known curve fitting technique. E.g. the signature 402 can becoded as a number of connected line segments of specific length andintersegment angle.

A motion trajectory estimation unit 206 calculates a motion trajectory400, on the basis of motion sensing means 304, or successive picturesfrom the camera 102. E.g. optical flow or block based motion estimationtechniques can be applied on two successive pictures to obtain a part401 of motion trajectory 400. The divergence of the motion vector fieldcan be used to estimate motion towards on object, i.e. typically alongan axis of the pointing device 101. On the basis of motion trajectory400, a signature generation unit 209 outputs a first motioncharacterizing signature s1. A signature identification unit 208identifies the first motion characterizing signature s I and link itwith command identification data ci, being e.g. a numericalrepresentation of a user interaction command intended by the user 100.The first motion characterizing signature s1 can be compared withsignature characteristics for different stored signatures in a signaturecharacteristics memory 232. E.g. if a roughly circular or even anyreturning movement is made, a stored circular signature template willyield a better match than a linear signature template. Geometrical andstructural curve matching techniques can be applied in the similaritycalculation.

Just like it is advantageous to have a DSP 200 which is able to learn toidentify new objects, it is advantageous if the DSP 200 can identify newmovements preferred by the user 100. Therefore the signature associationunit 214 might be comprised for providing signature association data SADto the signature identification unit 208. The signature association dataSAD may comprise associated signature features sa, e.g. a mathematicalrepresentation of the circular signature 412, and/or command relateddata cid, e.g. received from an apparatus sending its supported commandsas command data cd to the DSP 200. Signature templates in a signaturememory 222 can be obtained from a signature training unit 223 on thebasis of a first motion characterizing signature s2 from the motiontrajectory estimation unit 206 or from command data cd.

Since the environment of the user is very variable, e.g. the lightingchanges during the day, the identification of the object 130 or acommand, i.e. a signature 402, might sometimes be incorrect. Thereforethe DSP 200 might comprise an identification improvement unit 210. Theintended command is e.g. dependent on the object 130 pointed to, ormight be dependent on the command given previously, and so on. An objectis identified incorrectly if it is not present in the room the pointingdevice is residing in. Therefore room recognizing means 185 may bepresent, e.g. flashing LEDs. The user 100 can scan the room recognizingmeans 185 with the pointing device 101 when he enters the room. Apartfrom the camera 102, a photodiode might be comprised in the pointingdevice 101 for detecting the room recognizing means 185. Roomlocalization beacons 180, 181, 182 may also be present, so that the DSP200 can recognize to which part of the room the pointing device 101 ispointing. The beacons could be light emitting in case the camera 102 isto recognize them, but they could also e.g. emit electromagneticradiation. Objects present in the room can also aid in the recognitionof an object. E.g. if vase 170 is next to object 130, its recognitioncan aid in the recognition of object 130. Even analysis of the pasttrajectory of the pointing device can be used. If the pointing devicewas previously pointing to object 130 and it has been moved to theright, it should be pointing to apparatus 110. Taking all thisinformation into account, the identification improvement unit 210 cancalculate e.g. Bayesian probabilities or use fuzzy logic to arrive at amore certain identification of the object 130 and the intended command.

The user interaction information I outputted by the DSP 200 typicallycomprises control apparatus data ac, being e.g. a structure comprisingthe object identification data oi and the command identification dataci, or a new identifier based on those, identifying an intended commandfor a particular apparatus 110. The user interaction information I mayalso comprise picture data pd.

FIG. 3 schematically shows a pointing device 300—an embodiment of 101 inFIG. 1—according to the invention. Not all features shown need bepresent: a simple pointing device may only comprise a camera 302 ande.g. a display 316 for feedback. A number of buttons e.g. button 308might be present. This allows for giving certain commands only bepushing a button. Also similar movements of the pointing device 300 canbe used for two different commands, once with and once withoutsimultaneously pushing the button 308. A characteristic projector 320may be present, e.g. for projecting a laser spot towards the locationpointed to. The camera of the pointing device is capable of sending apicture to the external DSP 120, but it might also itself comprise theDSP 200. In this way the pointing device can also be used outdoors. Apointing device 103 not carried by the user 100 might also be used tomonitor an apparatus 110 or room. Apart from sensing the motion by meansof the camera, other motion sensing means 304 may also be comprised,e.g. a gyroscope or differential GPS. A programmable code generator 309and wireless transmitter 310 may also be present for sending a commanddirectly to an apparatus 110. Feedback means might be present forproviding the user with additional information. E.g. a light 312, asound production device 314, a force feedback means 306, and inparticular a display 316 might be comprised. The force feedback meanscan comprise a small moveable mass, which can simulate vibration or thetouching of an apparatus. The display can show e.g. the center of theregion pointed to, possibly superimposing a graphically generated crossin the center of the picture imaged. This can also be displayed on atelevision. Similarly the light 160 might be used to provide feedbackinstead of light 312. The feedback on the display 316 might show e.g.the motion trajectory 400 and how similar it was to previous motiontrajectories during the motion trajectory training phase, aiding theuser 100 to learn the DSP 200 to recognize a stable gesture. The display316 might also indicate what the DSP 200 sees by showing a picture aftera number of image processing steps, e.g. a picture containing detectededges, and comprising text naming the identified device.

User identification means 330 might also be present, e.g. a fingerprintsensor, or an input for a password or personal smart card. In this waythe DSP 200 can set itself to a mode of operation particular to the user100. The user might even give different commands depending on the roomhe is in. Speech recognition means 340 may also be comprised, tosupplement the commands given by movements of the pointing device 300.Games could be played by using the pointing device 101 as some kind ofinterface device to the real world.

The following embodiments give exemplifying combinations of technologiesinvolved in the present invention:

Embodiment 1

User interaction system, comprising:

an electrical apparatus;

a portable pointing device operable by a user for pointing to a regionin space;

a camera taking a picture; and

a digital signal processor, capable of receiving and processing thepicture, and capable of transmitting user interface information derivedfrom the picture to the electrical apparatus,

characterized in that the camera is connected to the pointing device sothat in operation it images the region pointed to.

Embodiment 2

User interaction system as in embodiment 1, wherein the user interfaceinformation comprises apparatus control data for controlling operationof the electrical apparatus.

Embodiment 3

User interaction system as in embodiment 1, wherein the digital signalprocessor comprises an object characterizing means for characterizing anobject or part of the object present in the picture of the region imagedby the camera, by providing first object characterizing features to acomprised object identification means for identifying the object, andwhich object identification means is capable of outputting objectidentification data from which the user interface information isconstructed.

Embodiment 4

User interaction system as in embodiment 1, wherein the digital signalprocessor comprises:

motion trajectory estimation means for estimating a motion trajectory ofthe pointing device and outputting a first motion characterizingsignature, a signature being a mathematical abstraction of the motiontrajectory; and

signature identification means for identifying the first motioncharacterizing signature and outputting command identification data,which represents a user interaction command, corresponding with thefirst motion characterizing signature, from which command identificationdata the user interface information is constructed.

Embodiment 5

User interaction system as in embodiments 3 and 4, wherein the digitalsignal processor comprises identification improvement means, which arecapable of further improving a probability that the object representedas object identification data, and user interaction command representedas command identification data, are more reliably identified based onpredetermined rules, yielding more reliable user interface information.

Embodiment 6

User interaction system as in embodiment 5, wherein the predeterminedrules comprise probabilistic calculation of the likelihood of an {objectidentification data, command identification data }—pair, taking intoaccount at least one of the following a priori known information units{room in which the pointing device resides, previous command issued byuser, statistical frequency that a user issues a particular command andtime of the day}.

Embodiment 7

User interaction system as in embodiment 3, wherein the digital signalprocessor comprises object association means for providing to the objectidentification means object association data—comprising at least one ofthe data entities being: associated object characterizing features andobject related data—, the object association data being derivable fromobject template data in object memory originating from at least one ofthe methods:

the object template data is obtained from object training meansperforming a predetermined calculation on second object characterizingfeatures outputted by object characterizing means; and

the object template data is derived from inputted object data.

Embodiment 8

User interaction system as in embodiment 4, wherein the digital signalprocessor comprises signature association means for providing to thesignature identification means signature association data—comprising atleast one of the data entities being: associated signature features andcommand related data—, the signature association data being derivablefrom signature template data in signature memory originating from atleast one of the methods:

the signature template data is obtained from signature training meansperforming a predetermined calculation on a second motion characterizingsignature outputted by the motion trajectory estimating means; and

the command template data is derived from inputted command data.

Embodiment 9

User interaction system as in embodiment 4, wherein the first motioncharacterizing signature is derived on the basis of successive picturesimaged by the camera at respective instances of time.

Embodiment 10

Pointing device for use in a user interaction system as in embodiment 1,characterized in that it comprises a camera and is capable of sending apicture to a digital signal processor.

Embodiment 11

Pointing device as in embodiment 10, wherein the pointing device iscapable of sending a picture to the digital signal processor, which iscapable of sending user interface information to an electrical apparatusbased on the picture.

Embodiment 12

Pointing device as in embodiment 10 wherein the digital signal processoris comprised in the pointing device.

Embodiment 13

Pointing device as in embodiment 10, comprising motion sensing means forsensing a motion trajectory of the pointing device.

Embodiment 14

Pointing device as in embodiment 10, comprising a characteristicprojector for optically projecting a characteristic pattern towards aregion pointed to.

Embodiment 15

Pointing device as in embodiment 10, comprising a programmable userinterface code generator and a wireless transmitter for transmitting thecode to the electrical apparatus.

Embodiment 16

Pointing device as in embodiment 10 comprising feedback means forfeedback of user interface information.

Embodiment 17

Electrical apparatus for use in a user interaction system as inembodiment 1, characterized in that interface means are comprised whichallow the electrical apparatus to send information about supportedcommands to a pointing device as in 1, based on an “identify supportedcommands” call of the pointing device to the electrical apparatus.

1. User interaction system, comprising: an electrical apparatus (110); aportable pointing device (101, 300) operable by a user for pointing to aregion in space; a camera (102) taking a picture; and a digital signalprocessor (120), capable of receiving and processing the picture, andcapable of transmitting user interface information (I) derived from thepicture to the electrical apparatus (1 10), wherein the camera (102) isconnected to the pointing device (101, 300) so that in operation itimages the region pointed to characterized in that the user interactionsystem further comprises a pattern creation means (116) to exhibit acharacteristic pattern that is created for identification purpose by thepointing device of the pattern creation means associated apparatus. 2.The user interaction system as in claim 1, wherein the digital signalprocessor (120) is further arranged to recognize the direction of thepointer relative to the localization beacon.
 3. The user interactionsystem as in claim 1, further comprising motion sensing means (304) forenabling estimating a motion or a motion trajectory (400, 410) of thepointing device.
 4. The user interaction system as in claim 1, whereinthe motion or the motion trajectory (400, 410) of the pointing device isestimated on basis of successive pictures imaged by the camera (102) atrespective instances of time.
 5. User interaction system as claimed inclaim 2, wherein the transmitted user interface information (I) includesat least one feature selected from the group consisting of motion speed,and motion direction of the pointing device (101).
 6. User interactionsystem as claimed in claim 3, wherein the transmitted user interfaceinformation (I) includes at least one feature selected from the groupconsisting of motion trajectory (400) of the pointing device (101) and acharacteristic signature derived from the motion trajectory (400) of thepointing device (101).
 7. User interaction system as claimed in claim 1,wherein the pointing device (101) further comprises feedback means forproviding the user with additional information.
 8. User interactionsystem as claimed in claim 7, wherein said feedback means is selectedfrom the group consisting of light (312), sound (314), a display (316)and force feedback means (306).
 9. User interaction system as in claim1, wherein the user interface information (I) comprises apparatuscontrol data for controlling operation of the electrical apparatus(110).
 10. Pointing device (101, 300) for use in a user interactionsystem as in claim 1, characterized in that it comprises the digitalsignal processor (120) and a camera (102) and wherein the pointingdevice is capable of sending a picture to the digital signal processor(120).
 11. Pointing device (101, 300) as in claim 10, comprising acharacteristic projector (320) for optically projecting a characteristicpattern towards a region pointed to.
 12. Pointing device (101, 300) asin claim 1, comprising a programmable user interface code generator(309) and a wireless transmitter (310) for transmitting the code to theelectrical apparatus (110).
 13. Electrical apparatus (110) for use in auser interaction system as in claim 1, characterized in that interfacemeans are comprised which allow the electrical apparatus (110) to sendinformation about supported commands to a pointing device (101, 300)based on an “identify supported commands” call of the pointing device(101, 300) to the electrical apparatus (110).